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ABSTRACT 


The Naval Postgraduate School has developed a competency model for the 
systems engineering profession and is implementing a tool to support “high stakes” 
human resource functions for the U.S. Army. A systems engineering career competency 
model (SECCM), recently developed by the Navy and verified by the Office of Personnel 
Management (0PM), defines the critical competencies for successful performance as a 
systems engineer at each general schedule grade level (GS-7 to GS-15). This 
foundational model is structured to support the individual needs of any Department of 
Defense organization and is made operable through the creation and implementation of 
tools to facilitate the management of systems engineering competencies at the local 
organizational level with traceability to the approved 0PM competencies. The Redstone 
SECCM Tool will allow documentation of system engineering competencies and 
assessment of individual and organizational development and training needs. This report 
documents the requirements analysis, system design, and system verification of the 
Redstone SECCM Tool, providing a path for implementation of the SECCM to support 
systems engineering human resource actions. 
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EXECUTIVE SUMMARY 


The Office of Personnel Management, with the assistance of the Naval 
Postgraduate School (NPS), developed a Systems Engineering Career Competency 
Model (SECCM) to meet the competency-based workforce management needs of the 
Department of Defense. A verified SECCM would assist in “high-stakes” human 
resource functions such as position description creation, selection, and promotion, as well 
as employee development. The SECCM is now the foundational competency model that 
any defense agency can use to describe work functions and competencies for the 
proficient execution of systems engineering functions. In order to maximize the utility of 
this document, NPS tasked a capstone group to explore the application of this SECCM at 
the organizational level. The capstone team chose to conduct the effort using a modified 
system engineering life cycle including a Stakeholder Needs and Requirements 
Definition process, a Design Definition process, and a System Verification process. 

The Stakeholder Needs and Requirements Definition process started by 
identifying relevant stakeholders. The stakeholders for this task include human resources 
departments, hiring managers, engineering supervisors of systems engineers, and systems 
engineering employees. A diverse set of stakeholders were identified across Aviation and 
Missile Research, Development, and Engineering Command (AMRDEC) who displayed 
a knowledge of systems engineering as well as a need for assistance in systems 
engineering career development or hiring-manager functions. Each stakeholder 
participated in an Institutional Review Board (IRB)-approved interview session intended 
to define the competency development needs. 

Once the data was gathered from the interviews, the capstone team used a Human 
Centered Design (HCD) method for interpretation and need definition. The capstone team 
transferred data to Post-it notes and sorted through answers to find common themes. 
Prom the data, the current state of the systems engineering career field is inconsistent, 
consisting of variations between services, training, and even between organizations. 
Different user personas were developed to identify potential users of the SECCM and aid 
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in identifying the underlying needs from the user feedback. A set of stakeholder needs 
was identified from the analysis, with the emphasis of this effort on 

• a need to refocus SE competencies from 44 to SE specific competencies and 

• a need for an adaptable career competency model that can be defined and 
managed locally at the organization level. 

The final step in the Stakeholder Needs and Requirements Definition process 
consisted of translating the stakeholder needs into actionable requirements. Using a 
brainstorming process that included affinity diagrams and trend analysis, a set of three 
primary requirements and three subordinate requirements were defined and utilized as the 
basis for the project. 

The Design Definition process began with alternative definition and selection. 
Multiple alternatives were considered to meet the stakeholder needs, including a user 
guide, custom software, a spreadsheet implementation, and a mobile phone app. Each 
design solution was intended to implement the SECCM into a more user-friendly product 
that could be tailored at the local level. The Analytic Hierarchy Process (AHP) was 
chosen as the method to complete an analysis of alternatives (AoA). Each alternative was 
weighed against three factors: ease of use, completeness of meeting all the requirements, 
and implementation time. After weighing each alternative against the three factors, a 
combination of the user guide and spreadsheet was chosen, which met the most 
requirements in the timeframe allowed. 

The spreadsheet solution, also called the Redstone SECCM Tool after the location 
of the capstone team, gives the user a step-by-step walkthrough on how to develop a local 
competency model. Each competency can have up to six user-defined proficiency levels 
and can be tied to the Army SECCM for “high-risk” human resource functions. After 
completion of the competency model, the model can be assigned to an individual 
employee to track the employee’s personal competency development. The user guide 
solution, now called the Redstone Systems Engineering Career Competency Model Tool 
User Guide, gives a step-by-step procedure on developing a competency model within 
the Redstone SECCM Tool and is a companion product to the tool. 
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The final step in the system engineering life cycle was the System Verification 
process. The stakeholders were once again asked a set of IRB-approved questions, this 
time on the quality of the tool and user guide. At the time of the event, the user guide had 
not yet been completed and therefore was not rated. The tool was given high ratings on 
most areas including meeting expectations, overall satisfaction, ease of use, and 
tailorability. After the verification event with the stakeholders was completed, metrics 
were an additional functionality added to the Redstone SECCM Tool. Metrics can now be 
tracked on all users of a single competency model within an organization to track 
organizational growth. Additional capabilities were identified but were unable to make it 
into the final version of the tool. These capabilities were included as a set of future 
recommendations for the tool, including as follows: 

• “Required at Entry” and “Important” discriminators among the competencies 

• weighting factors 

• a notes column 

• printable reports 

• historical progression tracking 

• additional graphing capability 

• tracking for specific positions or domains 


xvii 
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I. 


INTRODUCTION 


The need for qualified systems engineers within the DOD has been growing for 
many years. As the complexity of DOD systems evolves, so must the knowledge, skills, 
and abilities of systems engineers tasked with managing and directing complex systems 
in all stages of their life cycle (Government Accountability Office [GAO] 2015). This 
need is further reinforced by the publicity surrounding large program-cost overruns, 
coupled with the impacts of an ever-increasing number of failed programs (GAO 2014). 
These situations highlight the expanding need for properly trained, competent system 
engineers across the DOD. 

The Office of the Deputy Assistant Secretary of Defense for Systems Engineering 
(ODASD(SE)) is the focal point for increasing the development and use of organic 
systems engineering (SE) processes across the DOD (Under Secretary of Defense for 
Acquisition, Technology and Eogistics [AT&E] 2011). The ODASD(SE) is aggressively 
working to address a range of engineering challenges presented by the increasing 
complexity of emerging weapons systems (International Test and Evaluation Association 
[ITEA] 2016). The development of organic SE workforce competencies within the DOD 
is a crucial initiative needed “to improve engineering, test, and evaluation methods and 
tools; and to broaden its partnerships with commercial and defense industry, universities, 
and federally funded research and development centers that augment organic capabilities 
with critical talent” (ITEA 2016). To make significant progress in workforce 
development, the DOD still has work to do in order to define and develop many new 
areas for systems engineers, such as workforce training for requirements development 
and resource trade-offs (GAO 2015). 

The Department of Defense (DOD) utilizes a competency-based approach for 
workforce management (U.S. Office of Personnel Management [0PM] 2016a). This 
approach requires that the DOD assess the critical skills and competencies needed for its 
workforce both in the short and long term. The DOD must also plan and execute 
strategies for closing competency and skill gaps. 
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For the SE field, one method for executing this strategy in the long term is to 
create an SE occupational series code, which is currently nonexistent within the 0PM’s 
Position Classification Standards (0PM 2009). Given that the classification of positions 
for employees within government organizations is mandated under Title 5 of the United 
States Code (USC), this particular option has far-reaching implications. Eor short-term 
execution, the DOD has the option of using a competency model verified for “high 
stakes” Human Resource (HR) functions (Whitcomb et al. 2015; Khan et al. 2016). “High 
stakes” HR functions, such as selection, position description creation, and promotion 
require the use of a competency model verified in accordance with (lAW) the Uniform 
Guidelines on Employee Selection Procedures, mandated under Title 29 of the Code of 
Eederal Regulation (CER). These Uniform Guidelines are a set of Eederal laws and 
standards designed to assist employers, labor organizations, employment agencies and 
licensing/certification boards in complying with requirements that prohibit discriminatory 
employment practices (White et al. 2016). The lack of an occupational series for SE 
meant that the SE discipline did not have an existing competency model that could be 
used to meet the strategic implementation needed for either the long-term or short-term 
options. 

To help address the gap of a missing competency model for the SE field that 
would align with both the USC and CER, the Deputy Assistant Secretary of the Navy 
(DASN) for Research Development Test and Evaluation (RDTE) tasked the Naval 
Postgraduate School (NPS) with researching the SE discipline and developing a SE 
competency model for defense (0PM 2016a). This effort included research into existing 
SE functions and surveys of current systems engineers. The Systems Engineering Career 
Competency Model (SECCM) is the product of this DASN RDT&E sponsored effort 
between NPS and the 0PM. 

Due to the importance of verifying a competency model for use in “high stakes” 
HR functions, the 0PM conducted an effort to verify SE core competencies lAW the 
Uniform Guidelines. The 0PM utilized its Multipurpose Occupational Systems Analysis 
Inventory-Close-Ended (MOSAIC) methodology to verify the SECCM (White et al. 
2015). Through the MOSAIC process, the 0PM facilitated a series of Subject Matter 
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Expert (SME) panels to create a list of competencies and tasks to use as the basis for an 
Occupational Analysis Survey. The survey was given to a sample of the SE population 
across the Army, Navy, Air Eorce, and Missile Defense Agency (MDA). After receiving 
and analyzing the results, the 0PM identified critical competencies and tasks for DOD 
systems engineers (0PM 2016a). 

The SECCM is now the foundational competency model that any defense agency, 
such as the U.S. Army, U.S. Air Eorce, or U.S. Navy, can use as a resource to meet 
individual organizational needs, describing competencies and related work tasks needed 
for the proficient execution of SE job functions. The SECCM identifies critical 
competencies and tasks for systems engineers within the General Schedule (GS) pay 
grades, at the GS-07 to GS-15 grade levels. The SECCM can be tied directly to the USA 
Staffing Competency Network, assisting hiring managers in developing position 
descriptions and hiring actions (Defense Civilian Personnel Advisory System [DCPAS] 
2017). 

An NPS capstone project was organized to develop further a method for 
implementing the SECCM within an organization. A group of students, colloquially 
referred to as “Team Redstone,” within the NPS Systems Engineering Master’s Program 
selected this topic for their capstone project. The 0PM survey analysis and technical 
report for the U.S. Army was provided as a reference source for Team Redstone (0PM 
2016b), to develop an SE competency model implementation for the Aviation and 
Missile Research, Development, and Engineering Center (AMRDEC). This capstone 
project report presents the development process and the findings for this effort. 

A. PROBLEM STATEMENT 

The DOD SE field does not have an occupational series, leading to local 
organization variations in both the definitions and expectations of competencies for 
systems engineers (White et al. 2016). According to White et al. (2016), without a 
defined competency model for systems engineers that has been verified lAW Uniform 
Guidelines, there can be no consistency across DOD in creating position descriptions, 
assessing job candidates, hiring, providing a basis for employee performance, and 
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establishing career path models and associated development plans for SEs. Now that an 
OPM-validated competency model has been developed at the Army level, the need shifts 
to identifying how best to implement the SECCM locally in organizations and divisions 
that utilize SEs. The SECCM is a broad document that covers SE at the Army level and 
must be applied at the organizational level in order to reap the maximum benefit. 

B. PROJECT APPROACH 

This project is focused on the application of the SECCM for the Army, in 
particular the Army’s Aviation and Missile Research, Development, and Engineering 
Center (AMRDEC). Stakeholders were identified to represent ah the aspects of SE career 
competencies, and the stakeholder feedback and responses steered the project direction. 
These responses formulated what would become the needs and requirements of the 
project. 

The capstone team used a project management approach to the product 
development. This is documented as the Team Redstone Project Management Plan 
(PMP). The PMP is a formal document that describes how the project was managed, 
controlled, and executed during the project life cycle. The PMP contains the overarching 
guidance, procedures, schedules, and methods used for planning, monitoring, and 
controlling the overall project. 

C. REPORT ORGANIZATION 

This report is laid out based on Team Redstone’s tasks, which were centered on 
an SE life-cycle model approach. This approach was then tailored to the specific needs 
and challenges of the project. The sections of the report are: 

Chapter I, Introduction - Provides a background and problem statement to frame 
the project. 

Chapter II, Stakeholder Needs and Requirements Definition - Provides a detailed 
analysis of the stakeholder identification, needs analysis and results, and derived 
stakeholder requirements. 


4 



Chapter III, Design Definition - Provides an analysis of alternatives for potential 
design solutions and an explanation of the selected design solution. 

Chapter IV, System Verification - Provides a description of the verification effort 
associated with the selected design. 

Chapter V, Summary, Conclusions and Recommendations - Provides a summary 
of the problem, how the design solution was able to address the stakeholder needs, and 
the overall conclusions from the project effort, and a list of forward-looking 
recommendations to address other stakeholder needs. 
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II. STAKEHOLDER NEEDS AND REQUIREMENTS 

DEFINITION 


The foundation for implementing an SE process is a properly defined set of user 
needs. According to the International Council on Systems Engineering (INCOSE) 
Handbook, “Successful projects depend on meeting the needs and requirements of the 
stakeholders throughout the life cycle” (INCOSE 2015). The first step in the capstone 
project systems development process was the Stakeholder Needs and Requirements 
Definition process. Stakeholder needs definition was conducted to identify stakeholders, 
gather their needs, and analyze the information to determine their true needs to be used as 
a basis for the project developments. Often, there is also a discovery process that takes 
place over time to adequately understand the true needs of each stakeholder. To reduce 
the discovery time, the Human Centered Design (HCD) approach was used to ensure that 
stakeholders are involved in all phases of the design, that needs are adequately 
characterized, and that requirements are developed to meet the needs of the users. Eor this 
project, the Army SECCM was used as a point of reference to start the discovery process. 
This chapter discusses stakeholder identification, the HCD approach used to elicit 
accurate stakeholder needs, stakeholder needs analysis and results, and the feedback loop 
that resulted in a final set of stakeholder requirements based on the assessed needs. 

A. STAKEHOLDER AND NEEDS IDENTIFICATION 

The stakeholders for this task include human resources (HR), hiring managers, 
engineering supervisors who manage the daily tasking of systems engineers, and SE 
employees. The scientist and engineer career program (CP-16) proponency office within 
the Army Materiel Command (AMC) also served as a stakeholder, given its mission to 
develop career engineers. The CP-16 proponency office develops career plans and offers 
funding for career training, as well as project office personnel who manage the matrixed 
systems engineers, relying on their support to successfully field systems. Each of these 
stakeholders has some experience with the challenge of developing training plans, hiring, 
developing job descriptions, and defining career paths used by their employees to build 
competencies in SE. 
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Each identified stakeholder offers a unique perspective on the roles that systems 
engineers take within the Army as well as the necessary skills, training, and competencies 
expected. Each stakeholder participated in an interview session with open-ended 
questions intended to gain an understanding of their unique perspectives on SE within the 
DOD. Stakeholder identification was based on feedback from within each group 
members’ organization. In order to develop an Army SE competency model, stakeholders 
within AMRDEC, AMC, Program Executive Office Missiles and Space (PEO M&S), and 
Software Engineering Directorate (SED) were identified to provide a diverse perspective 
of the needs, capabilities, and shortcomings of tracking and understanding SE 
competencies within the Army. Each stakeholder is in a leadership position within his or 
her respective organization and each has expressed a sincere interest in furthering the 
development of an SE competency model. The power interest grid, shown in Eigure 1, 
was used to assess the level of influence shared among the selected stakeholders. 


1. AMRDEC ED* 

2. PEO M&S 1 

3. PEO M&S 2 

4. AMRDEC SED* 

5. AMRDEC WDI* 

6. CP-16 

7. MDA 

8. PEO M&S 3 

9. PEO M&S 4 



* Primary Stakeholders 


Interest 


Eigure 1. Power Interest Grid 


The project took an interpretive, qualitative approach to identifying and 


understanding stakeholder needs based on a predetermined set of interview questions. 
These interview questions were developed with the purpose of finding objective areas of 
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need based on identified issues and gaps that exist for SE within the Army. The NFS 
Institutional Review Board (IRB) determined that the interview questions were not 
human-subjects research. The questions, along with a copy of the 0PM “Army Systems 
Engineering Occupational Analysis,” were given in advance of the interviews to allow 
the stakeholders time to review the background information and make connections to the 
questions. Two members of the capstone project team conducted each interview; one 
team member served as the interviewer and the other assisted the interviewer by verifying 
that all of the questions were discussed and recording the stakeholder statements made 
during the interview. Interview notes are available in the appendix. 

B. HUMAN-CENTERED DESIGN APPROACH 

The standard SE approach was modified by integrating HCD methods. Detailed 
information relative to the human-system interactions that occur throughout the life cycle 
is given in the technical specification “Ergonomics of Human-System Interaction— 
Specification for the Process Assessment of Human-System Issues” (International 
Organization for Standardization [ISO] TS 18152 2010), while specific requirements and 
guidance relating to HCD are provided in the international standard “Human-centered 
design for interactive systems” (ISO 9241-210 2010). These documents help address 
human-centered issues that arise throughout a system life-cycle process and provide a 
human-centered perspective that is complementary to existing SE development process 
methodologies. Because HCD “is a creative approach to interactive systems development 
that aims to make systems usable and useful” (ISO 9241-210 2010), HCD forces the 
designer to empathize with the user and to help understand the needs while also creating 
a design that provides personal appeal. When done properly, a human-centered approach 
“fuels the creation of a product that resonates more deeply with an audience” (Thomsen 
2013). To gain greater contextual perspective on the stakeholders needs and work 
through to a solution, the capstone team performed HCD activities similar to those 
presented by IDEO (IDEO 2015). 

The HCD method adopted for this project follows an approach that aligns with 
technical life-cycle processes identified in ISO 15288 as stakeholder requirements 
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definition, requirements analysis, architeeture, design, implementation, integration, 
verification, transition, validation, operation, maintenance, and disposal (ISO 15288 
2015). The HCD stages that complement these processes include framing the problem, 
exploring the problem space, designing a solution through ideation and iteration, 
discovering or redefining the solution based on prototyping and feedback, and delivering 
the solution. The specific HCD steps and their comparable generic technical processes, as 
described in ISO 15288, are shown in Table 1. 


Table 1. HCD Stages 


HCD Stages 

Comparable Technical Processes 
(ISO 15288) 

Frame 

Stakeholder Requirements Definition 

Explore 

Analyze System Requirements 

Design 

Define Architecture 

Discover 

Design Architecture 

Deliver 

Implementation, Integration, Verification, 
Transition, and Validation 


Following these steps allowed for the explicit understanding of stakeholder needs 
by involving them throughout the entire developmental process. By employing an HCD 
methodology, the capstone team was able to view needs from the perspectives of the 
stakeholders while gathering critical information helpful in solving problems or issues 
exposed during this project. 

By working through the problem framing, exploration, ideation, iteration, and 
discovery stages, the capstone team was able to take a HCD approach toward the 
processes involved in the creation of the SECCM. For example, rather than attempting to 
define each of the 0PM SE competencies and include the entire set in the solution 
presented to stakeholders, the HCD design thinking approach resulted in questions that 
resonated with the stakeholders, such as “How does SE benefit the stakeholders 
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organization?” and “What specific competencies are required to be an effective systems 
engineer?.” 

Three issues discovered during this project revealed how HCD strategies were 
used to formulate effective mitigation strategies: 

Issue 1: Currently, there is no way to ensure that the stakeholders identified 
represent a large percentage of the SE population within Army or even across the 
organizations identified in this report. 

Mitigation strategy: The capstone team sought to overcome this by developing an 
adaptable and tailorable solution that is able to meet the needs of users not represented by 
the stakeholders selected for this project. 

Issue 2: The stakeholder questions may not have been designed correctly or do 
not clearly identify higher-level needs within the Army SE workforce. 

Mitigation strategy: By producing minimum viable products (MVP) that address 
the largest assumptions first and release to a small test audience (e.g., beta testing, smoke 
testing), additional insights can be gained. Eurther iterating and testing design alternatives 
based on these lessons learned will help to identify any higher level needs not addressed 
early on. 

Issue 3: There is risk associated with transcribing information from an interview, 
including loss of key points, lack of detail, and over-emphasis of nonessential 
information. 

Mitigation strategy: This is addressed in part by keeping an “open line of 
communication” with each stakeholder throughout the project to obtain additional 
feedback and clarification as needed to help further iterate to the proper solution. 

C. STAKEHOLDER NEEDS ANALYSIS 

The capstone team began framing the problem by capturing and characterizing 

stakeholder inputs. By transferring interview information to Post-it notes, the team was 

able to reduce the information to smaller bites for useful visual representation. The Post-it 

notes that captured each interview’s major points also listed the major takeaways from 
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each of the interviews. After correlating and sorting the information, the capstone team 
summarized the major points, presented in Table 2. Each cell represents an individual 
stakeholder’s input. 


Table 2. Major Interview Points 


Major Points 

• 

Competencies 1 - 19 are major SE competencies 

• 

All project offices act different 

• 

Need SE degree 

• 

SE upfront in the life cycle is most important 

• 

Need breadth and depth of experience 

• 

Need training AND experience 

• 

The government does not do SE 

• 

The government poorly performs in requirements engineering 

• 

Defense Acquisition University (DAU) training is a good start, but 
not enough 

• 

On-tbe-job training (with mentor) is required 

• 

Demonstrate experience 

• 

Eocus on skill set, not knowledge, then apply it. 

• 

Model small project success 

• 

Technical competence is most important 

• 

System experience as a whole 

• 

Breadth, depth in one area 

• 

On the job training, rotations 

• 

Aim at a job (ex. division chief), take the example and follow 
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Major Points 

• 

Locally managed competencies 

• 

Competencies are NOT binary 

• 

Need experience 

• 

No SE job series needed 

• 

Competency relevance to RDECOM 

• 

Army Career Tracker (ACT) training 

• 

Rotational training 

• 

Experience development 

• 

A systems engineer is a subject-matter expert 

• 

Experience (breadth) 

• 

Experience depth first 

• 

Eocally managed 

• 

Understand the entire life cycle 

• 

Define level of experience per grade 

• 

Need more training for leadership skills 

• 

On-the-job experience 

• 

Competency model will be helpful 


The capstone team then considered the differing viewpoints of the stakeholders as 
not every stakeholder defined a systems engineer in the same way. This was critical in 
defining and organizing the feedback. The outcome of this analysis allowed the capstone 
team to apply a filter to the stakeholders’ feedback. 
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From the stakeholder feedback, 0PM SECCM Report, and research of other 
existing competency models, the capstone team identified five characteristics that define 
the current state of the SE career field. Table 3 shows the current state. 

Table 3. SE Current State 

Ill-defined 

Inconsistent 

Large variation from organization to organization 
Variation in training requirements 

Differences among services (Army, Navy, Commercial, etc.) 


Given the findings, the capstone team considered the current state of the SE career 
field to be ill-defined as more than a few of the 0PM competencies were applicable to 
more than just systems engineers. The very definition of what a systems engineer is or 
does varies tremendously, even at local and organizational levels. To work toward 
correcting these issues with an end goal in mind, the capstone team proposes an “ideal” 
future state, as shown in Table 4. 

Table 4. SE Euture State 


Competency model exists and is used regularly 

Consistent definitions (skill sets, experience, etc.) exist and are referred to 
regularly 

Job series descriptions exist and are referenced to regularly 
Marketable skills/capabilities are developed in systems engineers 


Career paths (institutional training, on the job training, DAU, mentoring, 
ACT, etc.) exist and are used regularly 
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Finally, the capstone team documented all of the potential users of the SECCM 
and developed user personas to ensure consideration was distributed appropriately, 
shown in Table 5. A persona is simply a fictional representation of real users or 
stakeholders that might interact with the system or process, utilized with intention of 
focusing on the users within the HCD process. These personas try to capture as many 
use-cases as possible while still remaining relevant to the problem at hand. 

Table 5. User Personas 
Entry-level engineer 
Entry-level systems engineer 
Current systems engineer 
Candidate systems engineer 
Systems engineering director 
Human resources 
Hiring managers 


Once all of this information was considered, the capstone team met again solely to 
discuss all of the user persona information and develop the stakeholder needs. 

D. STAKEHOLDER NEEDS ANALYSIS RESULTS 

Using the combined INCOSE Systems Engineering Handbook, 4th edition, and 
HCD methods, individual replies from the stakeholders were grouped and categorized 
into similar needs based on affinity. As the capstone team uncovered common 
stakeholder interests, those items were refined and grouped to become stakeholder needs. 
Just as the stakeholder roles ranged from HR Management to SE practitioners, the replies 
received from the 17 questions varied in different topic areas and conclusions. The replies 
were reviewed and analyzed for any common issues or gaps. Some individual replies 
from the stakeholders have specific recommendations that can be used at the 
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requirements level. Those details are documented and archived for use later in the SE 
process. The capstone team analyzed the individual replies, concluding with a 
summarized list of stakeholder need statements as shown in Table 6. 


Table 6. Stakeholder Needs 


Need 

Index 

Needs 

N 1 

A single, agreeable definition for SE within the Army 

N2 

Need to condense SECCM-listed competencies from 44 to those 
specific to SE. 

N3 

Need an adaptable career competency model that can be defined 
and managed locally at the organizational level. 

N4 

Need training, experience, and education that encompass all phases 
of the system life cycle. 

N4.1 

Need a comprehensive list of on-the-job training that would 
strengthen the competencies of Army systems engineers. 

N4.2 

Need rotational assignments for systems engineers 

N5 

Need additional research to determine the benefit and feasibility of 
an SE job series. 


E. STAKEHOLDER REQUIREMENTS DEVELOPMENT 

A successful project depends on the ability of the SE team to translate the needs 
of the user into objective, demonstrable requirements. The purpose of the Stakeholder 
Needs and Requirements Definition Process as stated by INCOSE is to “define 
stakeholder requirements for a system that can provide the capabilities needed by users 
and other stakeholders in a defined environment” (INCOSE 2015). With stakeholder 
needs defined, a set of stakeholder requirements must be developed that 

• are intended to meet the stakeholder need(s), 

• clearly and concisely meet all parts of the need(s), and 
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• do not add capability outside of the need(s). 

In this early stage of requirements development, gathering input and defining the 
process are the two most important steps in developing the stakeholder requirements. By 
gathering input through direct interviews, the capstone team was able to view needs from 
the perspectives of each of the stakeholders. Once the stakeholder needs, the system life 
cycle, and problem statement were defined, the capstone team focused on utilizing 
techniques similar to the HCD ideation and iteration processes to transform stakeholder 
needs into requirements. The capstone team conducted brainstorming sessions to 
decompose needs into requirements that involved clustering of information into uniquely 
characterized groups (or buckets) to highlight key stakeholder needs. Trends within the 
needs were identified using affinity diagrams and primary needs were identified by these 
trends, with an influence factor relating to the stakeholder’s power versus their interest. 
By utilizing the power-interest grid in Figure 1, the capstone team was able to rank 
stakeholders based on their SE experience level and expected usage of the SECCM. 
Although many needs and subsequent requirements were identified during these 
processes, several were determined to be out of scope for this project based upon the 
capability limitations of the capstone team or project time constraints. Additional 
brainstorming conducted by the capstone team assessed select needs against candidate 
solutions and applicable verification methods, finding stakeholder needs N2 and N3, as 
defined in Table 6, to be within the scope of this project. The critical stakeholder needs 
identified during this phase were transformed into stakeholder requirements, as shown in 
Table 7, leading to key drivers to support the design phase of the capstone project. These 
requirements drove the design and development of a SECCM tool and user guide with the 
intended function to support the development of personnel and the tracking of 
organizational competencies. The design definition (selection and discovery) process is 
presented in Design Definition, Chapter III. 
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Table 7. Requirements Definition 


Need 

Index 

Requirement 

Index 

Requirement 

N1 

Out of scope 

N2 

R1 

The competency model shall he limited to or segregated hy 
system engineering specific competencies as defined hy Defense 
Acquisition Guide (DAG) Chapter 4 and INCOSE handbook. 


R2 

The in-use competency model shall he managed locally. 


R2.1 

The competency model shall provide measures to ohjectively 
scale progress. 

N3 

R2.1.1 

The measures shall he based on objective experience within that 
competency. 


R2.1.2 

The competency model shall define proficiency levels that 
indicate progress from lower to higher grade levels 


R3 

The competency model shall be tailorable at the organizational 
level and below. 

N4 

Out of scope 

N4.1 

Out of scope 

N4.2 

Out of scope 

N5 

Out of scope 
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III. DESIGN DEFINITION 


The Design Definition process, the second step in the capstone project systems 
development process, began once the capstone team felt confident that the requirements 
were objective and met the users’ needs. The purpose of the Design Definition Process 
according to INCOSE is to “develop, express, document, and communicate the 
realization of the architecture of the system through a complete set of design 
characteristics described in a form suitable for implementation” (INCOSE 2015). Design 
characteristics and design enablers related to each user-defined requirement are 
developed. Erom this set of characteristics and enablers, different design alternatives are 
brainstormed and assessed. The most appropriate alternative is then selected. The details 
of these design phases are presented in the following sections, including identification of 
alternatives, analysis of those alternatives within the project scope, summary of the 
results and conclusions from the analysis of alternatives, and description of the final 
design selected. 

A. ALTERNATIVES 

Eour alternative solutions were analyzed during this project and are described 

below: 

User guide - The first solution analyzed was a user guide and its purpose is to 
explain to a user how to translate the SECCM into a competency model specific to the 
user. The guide would be constructed such that a user could use it to completely 
understand the SE specific competencies and the more general competencies found in the 
SECCM and how to apply those competencies to their own organization. In order to keep 
this document from growing beyond the scope of the capstone project, the users are 
assumed to have a basic understanding of SE processes and competencies. 

Custom software - The second solution analyzed was a custom software program 
built from the ground up, which would create a local competency for the user. This 
software program would allow the user to create a local competency model derived from 
the SECCM. The software should be developed to be as user-friendly as possible in order 
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to reduce the user’s required knowledge of the SECCM. This solution requires that each 
competency be traced from a user’s competency to all applicable SECCM competencies. 

Spreadsheet - The third analyzed solution was a semi-automated spreadsheet to 
allow the user to create a set of local competencies based upon simple inputs. The 
spreadsheet would include “tool tip” type explanations of each step, and allow the user to 
create competencies with the necessary information at each step. This spreadsheet would 
be created in an existing, well-known program such as Microsoft Excel or Access in 
order to lower the coding time required and increase user familiarity with the interface. 

Mobile phone app - The final analyzed solution was a software application 
suitable for mobile phone usage. This app would allow an individual user to generate a 
competency model in a controlled environment but create a final product similar to the 
spreadsheet implementation. The app would require user input for the local competency 
while automatically generating the final competency product in the app. A competency 
model app example is the Naval Air Systems Command (NAVAIR) Career Guide Book 
available through the Apple app store. 

B. ANALYSIS OF ALTERNATIVES 

The analysis of alternatives (AoA) is an analytical process used to determine the 
solution that best meets the needs of the stakeholders. Also in consideration for this AoA 
is the time constraint of a solution that is achievable in the timeframe of a capstone 
project. 

The Analytic Hierarchy Process (AHP) was used to analytically compare the 
different alternatives to each other to determine the best solution to the user needs. The 
AHP consists of three steps using the alternatives and selection criteria outlined below 
(Goodwin and Wright 2014, 75-77). These steps provide a normalized result, which 
indicates the preference of each alternative relative to the others. A higher value means 
that that alternative meets more of the solution criteria than alternatives with a lower 
value. All relative comparisons for weighting were done according to the values shown in 
Table 8. 
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Table 8. Weighting Values 


Weighting Criteria 

Value 

Similar importance 

1 

Weakly more important 

3 

Strongly more important 

5 

Very strongly more important 

7 

Extremely more important 

9 


Each set of factors (in this case either the alternatives or the selection criteria) to 
be compared were placed in a matrix. The comparisons were performed row to column, 
meaning if the factor in the row was more important than the factor in the column, the 
corresponding value shown in Table 8 was entered. If the factor in the row was less 
important than the factor in the column, the inverse of the number given in Table 8 was 
entered. A priority vector was then calculated by normalizing for each factor by column 
total and then taking the average of each row. 

Selection criteria was defined using the AHP to select from the alternative to 
identify the final preferred solution. The following selection criteria was used to 
determine the final product output of the AoA: 

• Ease of use - The ease of use criteria is a determination of the ability for each 
user to utilize the solution to create the end product that meets the user needs. 

• Completeness - The completeness criteria is a determination of how many of 
the user needs the alternative will satisfy. 

• Implementation time - The implementation time is a determination of the 
ability for the alternative to be implemented in the time allotted for the 
capstone project. 

In the first step of the process, each of the selection criteria was weighted against 
each other to determine the level of dominance of one criterion over another in a matrix 
format. The “criteria” priority vector calculated for this matrix gives an understanding of 
the criteria which will have the highest impact on the analysis. 
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Next, each of the alternatives’ selection criteria was weighed against each other 
for a single selection criterion. This yields one matrix per selection criteria. The 
“alternative” priority vectors for these matrices show which alternative more strongly 
meets each criterion. Finally, each “alternative” priority vector was multiplied by the 
“criteria” priority vector for each criterion analyzed in the first step. The priority vectors 
were summed for each alternative to calculate the final total for each alternative. The 
alternative with the highest value best meets the criteria. 

Numbers above one (1) indicated that the item in the row was dominant to the 
item in the column. Numbers below one (1) indicated the item in the column dominated 
the item in the row. These were normalized, and then a priority vector was calculated to 
determine the relative weight of each selection criteria as shown in Table 9. The priority 
vector for all weighted comparisons was calculated by taking the relative weighting in 
each cell and dividing by the sum of the column. This is the new cell value. Then, these 
cell values are added to get the priority vector. 


Table 9. Weighted Comparison for AoA Selection Criteria 



Ease of 
Use 

Completeness 

Implementation 

Time 

Priority 

Vector 

Ease of Use 

1.00 

0.20 

0.33 

0.12 

Completeness 

5.00 

1.00 

0.20 

0.28 

Implementation 

Time 

3.00 

5.00 

1.00 

0.60 


The implementation time was evaluated to be approximately 60% of the total for 
the selection criteria. This means that the majority of the weight of the selection criteria 
was whether the alternative could be completed in the time allotted. Approximately a 
quarter of the weight was the capability of the alternative to meet the user needs, and the 
remaining 12% of the weight was given to the ease of use of the alternative. 
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Next, each criterion was evaluated independently by the alternatives. For each 
criterion, the capstone team determined the strength of importance for the alternatives 
against each other. These values were ranked according to the weighted values given, 
with separate comparisons made for the ease of use criteria as shown in Table 10. 


Table 10. Weighted Comparison of Alternatives for Ease of Use 


Ease of Use 

User 

Guide 

Custom 

Software 

Spreadsheet 

Mobile App 

Priority 

Vector 

User Guide 

1.00 

0.14 

0.20 

0.33 

0.05 

Custom 

Software 

7.00 

1.00 

3.00 

0.33 

0.35 

Spreadsheet 

5.00 

0.33 

1.00 

5.00 

0.34 

Mobile App 

3.00 

3.00 

0.20 

1.00 

0.26 


The analysis showed that the user guide was the lowest-rated alternative for ease 
of use, not being rated higher than any other alternative. The custom software and 
spreadsheet were rated equal to each other and more useful than the app. The results of 
this analysis support this conclusion. Review of the priority vector column showed that 
the custom software and spreadsheet were approximately equal in meeting the ease of use 
criteria. 

Table 11 shows that the mobile app was rated the lowest for completeness, while 
the user guide was rated higher than any other criteria. This was supported by the priority 
vector, which showed the user guide as being twice as likely to meet the completeness 
criteria as the next highest alternative, custom software. 
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Table 11. Weighted Comparison of Alternatives for Completeness. 


Completeness 

User Guide 

Custom 

Software 

Spreadsheet 

Mobile 

App 

Priority 

Vector 

User Guide 

1 

3 

3 

9 

0.52 

Custom Software 

0.33 

1 

3 

5 

0.27 

Spreadsheet 

0.33 

0.33 

1 

5 

0.16 

Mobile App 

0.11 

0.2 

0.2 

1 

0.05 


Table 12 shows that the custom software was rated lowest for the implementation 
time criteria. The mobile app was also rated low, beating only the custom software. 
Again, the priority vector bears this out, with custom software very low on the rating to 
meet this criteria, with the spreadsheet highest and the user guide next. 


Table 12. Weighted Comparison of Alternatives for Implementation Time 


Implementation 

Time 

User Guide 

Custom 

Software 

Spreadsheet 

Mobile 

App 

Priority 

Vector 

User Guide 

1.00 

3.00 

3.00 

0.33 

0.29 

Custom Software 

0.33 

1.00 

0.14 

0.33 

0.06 

Spreadsheet 

0.33 

7.00 

1.00 

5.00 

0.39 

Mobile App 

3.00 

3.00 

0.20 

1.00 

0.26 


C. AOA RESULTS 

The priority vectors for each alternative were combined into the final results 
shown in Table 13. The user guide and spreadsheet options were nearly identical in their 
final weighted result based upon the analysis. The mobile app and custom software were 
well behind the top two in ability to meet the specified selection criteria. 
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Table 13. Analysis of Alternatives Results 


User 

Guide 

Custom 

Software 

Spreadsheet 

Mobile 

App 

0.33 

0.15 

0.32 

0.20 


D. AOA CONCLUSIONS 

The AHP revealed that the spreadsheet and user guide solutions were very closely 
weighted in terms of the selection criteria. Based upon this result, a hybrid approach was 
selected that combined a spreadsheet solution (to guide users to the creation of their own 
local competency model) with the deeper explanation and definition provided by the user 
guide. To accomplish the hybrid solution, the user guide was tailored to the usage of the 
spreadsheet and not to the SECCM exclusively. 

E. DESIGN SOLUTION 

A Microsoft Excel-based SECCM tool, named Redstone SECCM Tool, along 
with a user guide, was selected as the design solution that would most efficiently meet the 
stakeholder needs. The tool provides a wizard-based, user-friendly interface for creating 
competency models that are intended to be traceable back to the 0PM SECCM 
competencies. The tool provides flexibility by allowing the user to either generate 
organizational models for dissemination or develop employee competency models. The 
employee models can be used within the local organizations for developing training 
plans. The Welcome Menu, shown in Eigure 2, provides primary entry points for using 
the tool to create or update the competency models. 
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Welcome 





Naval Postgraduate School 

Monterey, Californio 


New Base ] Create New Base Model 


New Employee 


Create New Employee Model 


Update Base Update Model 


Update Employee Update Employee Progress 



j .Metrics Generate Metrics 




Systems Engmeeiing Competency Model Tool 


Figure 2. Welcome Menu 


The “Create New Base Model” button initiates a wizard for creating a base 
competency model. The base model provides a list of competencies that can flow down 
from the organization level to the local level or simply be created at the local level. The 
model provides the ability to create new local competencies, develop proficiency levels 
for each competency, and then link them to a list of the 0PM competencies. The base 
model is the local organization list of SE competencies that can be used to create job 
descriptions for managing employees within a single organization; therefore, the base 
model may contain more competencies than a standard employee model. The model 
currently only allows development using the bottom-up approach with either the 
organizational or local competencies and linking them up to the 0PM competencies 
establishing traceability back to the 0PM model. The traceability to the 0PM 
competencies supports the development of job announcements compliant with the 0PM 
hiring policies. An example of a base model is shown in Figure 3. 
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Figure 3. Example Competency Base Model 


The base model in Figure 3 shows the local competency on the left and the 
corresponding 0PM competency on the far right, which shows the linkage between them. 
The figure identifies the description required at each corresponding level of knowledge 
for each competency. The model provides additional descriptions when the mouse hovers 
over the buttons, including functional details on the operation of each button. 

The “Create New Employee Model” button initiates a wizard for creating an 
employee model constructed from the base model. The employee model is a replication 
of the base model that is linked to that employee, but can also be tailored to the specific 
employee. The model supports the development and monitoring of employee progress for 
developing competency-specific knowledge. Figure 4 is an example of an employee 
competency management model. The blue-shaded cells represent the current state of 
competency knowledge for the employee. Additional controls, such as the example 
controls for an employee model shown in Figure 5, are embedded in the worksheet that 
allow the user to perform several functions for model management. 
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Figure 4. Example Employee Competency Management Model 


Save 


Close 


Add New Competency 


Update Progress 


SEP Development 


Remove Competency 


Eigure 5. Example Employee Model Controls 


The controls perform various standard functions such as saving the model, 
updating employee progress, and adding and removing competencies. The “Close” 
function closes the worksheet and restarts the tool welcome menu. The controls also 
provide the ability to add or remove competencies as the project evolves or the employee 
changes responsibilities. The “Update Progress” control advances the blue cells from left 
to right in order to indicate the current competency level and recycles once the end is 
reached. The “Update Model” button in Eigure 2 updates the user selected model using 
controls similar to those shown in Eigure 5. The controls in Eigure 2 do not have the 
update progress as that is only available when in the employee model. 


28 


































The “Update Employee Progress” button updates the employee model using the 
controls shown in Figure 6. The Update Employee Progress selection menu shown in 
Figure 6 provides the capability for exporting and importing the employee model once 
the employee is selected. The export capability allows the model to be provided to a 
subordinate organization for utilization or to the employee for updating relevant 
information in the model. The import capability allows the user to import the model once 
the update is complete. The export and import controls provide the ability to provide 
models for use by other organizations and to consolidate to models as necessary to 
maintain configuration control. 



Figure 6. Update Employee Model 

The tool provides a straightforward solution for creating competency models for 
use at the organizational or local level. The resulting models provide several important 
capabilities like identification of local competencies, assessing employee knowledge, 
identifying areas for focused training, and linking back to the 0PM competencies. The 
next step is System Verification to verify the model with the stakeholders, presented in 
Chapter IV. 
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IV. SYSTEM VERIFICATION 


This chapter discusses the system verification process applied to the SECCM 
project. Topics covered include the verification methodology, the verification results, and 
improvements made to the Redstone SECCM Tool as a result of the verification process. 
This is the third and final step of the capstone project systems development process. 

A. METHODOLOGY 

The methodology used to verify user needs and requirements consisted of face-to- 
face meetings with the main stakeholders to provide a walkthrough of the Redstone 
SECCM Tool and its capabilities, and then a method to capture metrics for the tool 
through the use of survey questions. The NFS IRB determined that the survey questions 
did not constitute human subjects research. Any issues or recommended changes are 
addressed based on the feedback from the meetings and the survey metrics. Some 
requirements are beyond the scope of this capstone project. With requirements known 
and discussed, the capstone team developed objective product metrics that would be 
asked to the main stakeholders with the intention of assessing how well the Redstone 
SECCM Tool met the needs that were laid out earlier in the project process. The metrics 
used to evaluate the system had a scale rating of 1 to 5 with 5 being the most satisfied. 
The metrics included evaluation in the following areas: 

• customer expectation 

• likelihood to recommend to others 

• overall satisfaction 

• ease of use 

• quality of help or user guide 

• tailorability 

• tool organization 

• ability to meet stakeholder needs 

• capability to help develop a training plan 
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Due to time constraints of the stakeholders, the capstone team demonstrated how 
the Redstone SECCM Tool was to be used by providing an example of how competency 
data is entered into the tool and displayed. The focus was on the bottom-up approach, 
which is defined as the method of creating a competency model by starting with defined 
local competencies and then tying them back to higher level Army competencies. Since 
most projects and organizations know the competencies required of their personnel, the 
bottom-up approach was the logical starting point for the tool. The capstone team 
demonstrated to the stakeholders how the Redstone SECCM Tool functions and how the 
supervisor can use the tool for personnel development and position descriptions, versus 
how the employee could use the tool to guide their career path. The demonstrations were 
consistent among stakeholders as to not skew the results. Due to time limitations, the 
stakeholders were not given a user guide prior to the initial demonstrations. 

B. VERIFICATION RESULTS 

The stakeholders were eager to see the results of the Redstone SECCM Tool 
development and each stakeholder felt more should be done to evaluate and assess 
system engineering competencies, so this tool concept was considered a step in the right 
direction. Three main stakeholders were used for the verification of the Redstone 
SECCM Tool due to their direct interest and expected usage after this capstone project is 
complete. A summary of each stakeholder’s assessment is provided in the following 
paragraphs. Eor anonymity, the names Stakeholder 1, Stakeholder 2, and Stakeholder 3 
will be used in this report. 

I. Stakeholder I 

During the first verification event, the capstone team described the project 
problem and how the Redstone SECCM Tool was designed to help solve that problem. 
The verification event included a demonstration of the tool; unfortunately, there were 
some minor technical difficulties, but the overall goals of the tool were on display. 
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a. Stakeholder 1 Observations 

• The stakeholder was more interested in how this could be used in hiring and 
promotion than in personnel tracking. 

• The stakeholder wanted to understand how an employee has progressed over 
time. History and progression were also major factors. 

• The stakeholder requested a notes column. 

Running through this effort demonstrated the importance of the top-down method. 
If AMRDEC-defined competencies existed, it would be easy to flow these down to 
organizations and divisions. 

b. Stakeholder 1 Recommendations 

• Pre-populated level fields 

• Ability to compare two (or more) employees graphically 

• More reporting options 

• Printable version of employee fields 

• Historical progression 

The use of competencies during mid-career planning seems to be underutilized 
among organizations and employees. The Redstone SECCM Tool is designed to assist in 
career planning and many of the recommendations above would aide in that endeavor. 

2. Stakeholder 2 

Stakeholder 2 was not able to attend the In-Process Review #2 so the stakeholder 
was unfamiliar with the progress of the tool and its capabilities to this point. During 
initial discussions the capstone team explained that the tool is a “framework” for entering 
the desired competencies determined by the user, not that there is a default set of detailed 
competencies or library function currently embedded in the tool. The capstone team 
explained that every user will likely have a different purpose and likely a different set of 
competencies considered of value. 
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a. Stakeholder 2 Observations 

• Entering the competencies allowed the user flexibility and adaptability of use, 
which the stakeholder found useful. 

• Stakeholder believed traceability to the 44 0PM competencies was a useful 
capability provided by the tool. 

b. Stakeholder 2 Recommendations 

• Provided positive feedback. 

• Felt the tool could be useful for existing projects. 

• Requested a copy of the Redstone SECCM Tool for further evaluation. The 
user guide was not available at the time of the interview, so it could not be 
utilized or reviewed during the interview. 

3. Stakeholder 3 

Stakeholder 3 was familiar with the Redstone SECCM Tool basics, as a similar 
tool is used in that office for employee assessment and training needs. Because 
Stakeholder 3 participated in the In-Process Review #2, the stakeholder was familiar with 
the capstone project scope and tool progress to date. 

a. Stakeholder 3 Observations 

• Indicated a desire to use the tool more within the management and team role 
versus at the employee role. 

• Indicated tool was geared toward individual usage, but not designed for 
collection of summary competency data to be displayed graphically. 

b. Stakeholder 3 Recommendations 

• Recommended that the tool include a summary roll-up to be used by the 
manager to demonstrate organizational effectiveness, and to determine if skill 
sets are growing or eroding over a period of time 

• Noted that it was important for the tool to be used as dialogue between 
employee and management, and that it should not be used for performance 
ratings or promotions. 

Presented in Table 14 are the results from the tool assessment performed by the 
stakeholders, which indicate that the original requirements (shown in Table 7) have been 
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satisfied. All users scored the tool with a three or higher in every category. Following 
completion of the tool and user guide, each stakeholder requested a chance to perform a 
follow-up review in order to reevaluate the criteria that was marked to be determined 
(TBD). 


Table 14. Tool Assessment Summary 


Product Metric 

Stakeholder 

Stakeholder 

Stakeholder 

(1-5, with 5 most satisfied) 

1 

2 

3 

Customer Expectations 

5 

5 

3 

Likelihood to recommend to others 

4 

TBD 

3 

Overall Satisfaction 

4 

5 

3 

Ease of use 

5 

5 

5 

Quality of help or user guide 

TBD 

TBD 

TBD 

Tailorahility 

5 

5 

5 

Tool organization 

4 

TBD 

5 

Meet the need 

3 

TBD 

4 

Help develop a training plan 

4 

3 

4 


Based on the feedback received from the users, a list of modifications was 
documented for future application. Not all enhancements could be made due to academic 
time constraints to complete the project, but these other enhancements are documented as 
future considerations that are recommended to be made in any future tool developments. 

C. MODIFICATIONS TO REDSTONE SECCM DEVELOPMENT TOOL 

Each stakeholder had a demonstration of the Redstone SECCM Tool via the 
verification process. Along with the tool assessment survey, valuable feedback was 
received from each stakeholder on further modifications to the tool that could make it 
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more beneficial to their divisions. The potential modifications as functions per the 
stakeholder feedback were as follows: 

• Output necessary information for job descriptions/announcements. 

• Assign a value of “Required at Entry” for specific competencies. 

• Assign a value of “Importance” for specific competencies. 

• Perform top-down competency development. 

• Assign weighting factors for each competency. 

• Track domain expertise for each competency (e.g. RAM, Failure Modes 
Effects and Criticality Analysis [FMECA], Missiles). 

• Add a “Notes” column within the competency model view. 

• Trace specific competencies or proficiency levels to rank/grade. 

• Output model into a printable report. 

• Create different types of graphical views of competencies and of employee 
progression. 

However, due to the constrained time line of the capstone project, the team had to 
decide which ones could actually be developed and integrated. The two major factors for 
this decision were development and integration timeline, as well as which capability 
provided benefit to the most stakeholders. Considering these two factors, the capstone 
team was able to include limited graphical view capability to the tool. Only two types of 
graphical views are currently available within the tool. The “Generate Metrics” button 
generates knowledge-level metrics for employee(s) depending on which metric type is 
selected from the menu shown in Figure 7. The metrics have two types: generating 
graphics for a single employee or generating graphics for all the employees with models 
in the spreadsheet. Results for the Employee Status example for a single employee are 
shown in Figure 7. The Competency Summary Rollup generates model specific metrics 
for all employees, with example results shown in Figure 8. The metric outputs can be sent 
to Microsoft PowerPoint for further development. “Generate Metrics” requires that the 
employee models be created using the correct base model and have progress completed 
before metrics can be generated. 
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Figure 8. Example Summary Rollup Metrics 

While the current schedule constrained further development of more graphical 
displays, the baseline architecture was built. Development time was spent to develop the 
architecture to properly store model data. With this architecture in place, development 
time for additional graphical views is minimized. 
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V. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 


The capstone project for utilizing the 0PM SECCM and developing a 
competency model solution for the AMRDEC stakeholders yielded several results. The 
initial expectations from the project problem statement changed throughout the initial 
user needs analysis and system development. The original project was intended to be the 
application of the SECCM at the organizational level with the focus revolving around the 
creation of an SE job series or the creation of the associated training requirement for the 
job series. The resultant product of a Redstone SECCM Tool User Guide and Redstone 
SECCM Tool fulfills the need as defined in the problem statement. 

A. SUMMARY 

The project began with the 0PM SECCM model and resulted in a competency 
model tool for addressing the needs as defined by the focused stakeholders group. The 
identification of stakeholders proved to be challenging and originally started with 
representatives from AMC, MDA, PEO M&S, and AMRDEC. The capstone team made 
the decision to concentrate on the three AMRDEC stakeholders due to the inconsistent 
stakeholder availability of the other organizations. As the capstone team members were 
employees of the AMRDEC, it presented an opportunity to focus the project on a 
common organization. 

The capstone project utilized the HCD methods throughout the SE process 
approach. The HCD approach makes the involvement of the stakeholders an essential 
part of the development process to drive the design toward the solution that met the 
stakeholders’ needs. The needs analysis process results indicated the current state of the 
SE career field definition at the AMRDEC to be ill-defined and inconsistently applied 
across different organizations. The needs derived from this analysis were developed using 
the stakeholder viewpoints and user personas resulting in seven stakeholder needs. 

The capstone team analyzed the needs to develop the associated stakeholder 
requirements. It was at this point that the needs evaluation identified specific needs that 
meet the intent of the project scope. The primary scope driver for the project was the 
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limited timeline of three academic quarters to complete the project while meeting the 
capstone team’s goal of producing a result that met the stakeholder’s needs. Based on 
team discussion and the identified constraints, the project was reduced from the seven 
identified needs down to two needs that would become the focus of the requirements 
analysis. The remaining five needs identified in Tables 6 and 7 were determined to be out 
of scope for this project. The two needs were decomposed into six requirements that were 
used to drive design alternatives during the design phase. The design solution chosen by 
the capstone team was evaluated against these requirements during the verification phase. 

The design phase took the requirements and developed characteristics and 
enablers for brainstorming design alternatives for selection and resulted in four design 
alternatives. The AoA evaluation was accomplished by using the AHP for analytically 
comparing the four design alternatives. The analysis determined that the two best options 
were the Redstone SECCM Tool and Redstone SECCM Tool User Guide. The Redstone 
SECCM Tool provides a semi-automated tool to develop local competencies for an 
organization, and optionally for individual personnel, traceable back to the 0PM SECCM 
model. The User Guide is a document that describes how to use the Redstone SECCM 
Tool to develop the local and user competency models. The decision to combine the 
Redstone SECCM Tool and the Redstone SECCM Tool User Guide improved the ability 
of the users to better understand the Redstone SECCM Tool and offers suggestions on 
how to apply the resulting models, thereby further improving the solution integration. 
Combining the tool with the user guide alleviated some of the original stakeholder 
concerns about the complexity and vagueness of the 0PM model and met the needs of 
the user more effectively than either could alone. 

The system verification technical process establishes traceability of the design 
solution back to the stakeholder requirements. The Redstone SECCM Tool and Redstone 
SECCM Tool User Guide verification consisted of providing a beta version of the 
products for stakeholder evaluation. Due to schedule constraints, the initial stakeholder 
evaluation of the Redstone SECCM Tool did not include the Redstone SECCM Tool 
User Guide as part of the evaluation. The updated Redstone SECCM Tool and Redstone 
SECCM Tool User Guide will be provided for an extended stakeholder evaluation once 
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the initial comments are incorporated. The three primary stakeholders evaluated the 
Redstone SECCM Tool using metrics traceable to the stakeholder requirements to 
determine stakeholder acceptance. The initial feedback resulted in scores for the 
Redstone SECCM Tool that indicated that requirements were met. The three primary 
stakeholders provided input for follow-on tool development. 

B. CONCLUSIONS 

The main conclusion is that there is a significant user need for a product like that 
of the Redstone SECCM Tool. The capstone team found an overarching need for 
identifying competencies, determining knowledge level, and documenting the 
deficiencies within the AMRDEC SE workforce. The stakeholders consistently 
commented that the Redstone SECCM Tool provided an effective framework for 
developing these models and identifying deficiencies. The tool also provides linkage back 
to the 0PM SECCM model competencies, supporting the ability to produce the job 
descriptions and announcements for hiring skilled systems engineers. Overall, the end 
products were well received. 

The stakeholders almost unanimously agreed that the 0PM model was too broad 
and too vague and encompassed too many competencies to be effective. Upon further 
investigation into the 0PM competencies, it was determined that more than a few were 
applicable to more than just systems engineers. Eor example, many members in the 
workforce beyond systems engineers need competencies like acquisition and 
configuration management, to name a few. This was a driver in the design criteria for the 
product with the resulting Redstone SECCM Tool having all the 0PM model 
competencies available but is tailorable to only those applicable to the local organization. 
That was important for the stakeholders, allowing local organizations to identify only 
those competencies specific to their programs and track progress against them. The 
additional benefit of providing the linkage to the 0PM competencies made the ability to 
produce job descriptions and announcements compliant with the 0PM policies much 
easier. 
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A third finding from the project was that the stakeholders recognized the need for 
identifying local competencies and tracking the knowledge growth of their employees. 
Several stakeholders expressed the need for comparing employees to assess the 
organization knowledge gaps and overall growth. All three primary stakeholders needed 
to effectively track the organization knowledge growth but internally developed three 
different metrics. One stakeholder expressed an overall need to assess the individual 
knowledge gaps so that effective training plans could be created to address those gaps. 
Another stakeholder expressed the need for demonstrating the progression of 
organizational knowledge over time, which characterizes the broad knowledge status as 
the organization grows and changes. The final stakeholder was focused on how the tool 
could assist in hiring and promoting. The differences seen in only three stakeholders 
highlight the range of competency model usage and emphasizes the challenges associated 
with implementation of a single approach for SE within the DOD. Additional stakeholder 
insights would help place a greater emphasis on identifying and defining the comparable 
SE competencies across the DOD. 

The needs analysis effort presented a clear picture of the current state of SE and 
how it is just starting to solidify with AMRDEC local organizations. This is potentially 
due to limited detailed guidance from DOD on SE and the learning curve associated with 
how to apply SE at the local levels. In the absence of additional details, organizations 
have started to implement SE concepts and principals within their organizations to 
differing degrees. Systems engineering consists of widely varying roles from the systems 
engineer who is expected to be the system subject matter expert to being system technical 
integrator. These differing roles may produce confusion within the SE community, 
limiting the ability for the movement needed to expose them to different aspects of SE 
that in turn limits career growth. This project provides the initial step of creating a 
framework for creating local competencies traceable back to the 0PM model within a 
highly flexible tool. The Redstone SECCM Tool provides the linkage of the 0PM 
competencies to the local organizations competencies. The Redstone SECCM Tool fills 
the gaps identified by the stakeholders during the early project analysis while providing 
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the traceability back to the 0PM model establishing the initial step for developing the 
DOD SE workforce. 

Overall the Redstone SECCM Tool was a success as identified by the 
stakeholders and represented by the high metric scores. As the project concluded, the 
stakeholders expressed considerable appreciation for how well the tool solved the 
underlying need of developing organization’s competencies and tracking employee 
progress against them. The capstone team was surprised how well the tool was accepted 
and the comments about how they planned on using it for supporting their organizations 
in different ways. The resulting tool usage is expected to be developing organization 
competency models and supporting the organizations in recruiting and developing 
existing systems engineers using the model data. 

C. RECOMMENDATIONS 

The Redstone SECCM Tool was developed as a prototype with the intention of 
applying the 0PM Army SECCM at the organizational level. The limited time to develop 
requirements, assess design alternatives, and develop and verify the Redstone SECCM 
Tool meant that some of the capabilities that would be beneficial in the tool had to be left 
out for future revisions. Additionally, as stakeholders and other potential users within the 
target organization began to see the utility of the Redstone SECCM Tool, other secondary 
capabilities were requested to assist the user track and manage competency models for 
the organization and personnel. This section summarizes most of these capability 
additions for future consideration. Each recommendation is presented along with a brief 
description; no weighting or ranking are given to any of the recommended changes, as 
the goal was to summarize all future capabilities desired. 

1. Recommendation: Job Description and Announcement Output 

The user requested the capability to output necessary information for job 

descriptions or announcements. One of the benefits of the Redstone SECCM Tool is that 

it links local competencies for a given position to the 0PM Competency Model used in 

USA Staffing to create job announcements. The capability to take an existing model and 

easily output the necessary information for USA Staffing to create a job announcement 
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would strea ml ine the data creation process for hiring managers, reduce errors, and 
provide more consistency. 

2. Recommendation: “Required at Entry” Competency Discriminator 

The stakeholders requested the capability to assign a value of “Required at Entry” 
for specific competencies. This capability would provide prospective systems engineers 
with insight into the necessary competencies before applying for a position. This ties 
back to the USA Staffing job announcement creation process as well. 

3. Recommendation: Discriminator for More “Important” 

Competencies 

Similar to the “Required at Entry” recommendation, the stakeholders would like 
to assign a value of “Important” to specific competencies. Some local competencies can 
link to multiple 0PM competencies. By allowing specific 0PM competencies to be more 
important to a local competency, this would streamline the job hiring and announcement 
process with USA Staffing and add a layer of clarity when many competencies are 
selected. 

4. Recommendation: Top-Down Approach for Competency Model 
Development 

The top-down competency development would include all local competencies 
within an organization in a drop-down box as well as built-in linkage between 0PM 
competencies and local competencies. The Redstone SECCM Tool was originally 
envisioned to contain both a top-down development capability as well as a bottom-up 
capability, but the top-down capability was determined to be too time consuming for the 
effort. The top-down development capability would require a database of all existing 
local competencies from which to select on new competency models. This was not 
feasible given the timeframe of the effort and the size and footprint of the organization. 
Additionally, each local competency should already be linked to the 0PM competencies 
to reduce the burden on each new competency model that is developed. 
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5. Recommendation: Weighting Factors 

One stakeholder recommended a method to weigh competencies against each 
other, to add an additional layer of objectivity to the model. This would benefit more in 
the hiring process than in employee development. This would also give the organization 
an opportunity to develop critical competencies within the organization. Along with the 
“Important” competency discriminator, this demonstrates that the stakeholders are 
interested in which competencies are most important both to the organization as well as 
to the employee. 

6. Recommendation: Domain Experience Tracking 

One stakeholder requested the capability to track domain experience for each 
competency (e.g., RAM, FMECA, Missiles). Within DOD, there are multiple domains of 
experiential knowledge that are gained depending on the product type. Some situations 
may be easier than others to translate skills between product types. The ability to track 
not only a competency but also the domain or domains in which proficiency is gained 
would be a valuable addition to the Redstone SECCM Tool. 

7. Recommendation: Notes Column 

As employees are given opportunities to further their advancement in a particular 
competency, it would be beneficial to take notes within the tool based on training taken, 
goals, or other administrative items that would not be shown within the tool already. 

8. Recommendation: Trace Competency to Rank or Role 

All stakeholders recommended that the tool be able to trace specific competencies 
or proficiency level to rank or grade. One in particular wanted to see roles and jobs being 
tracked to competencies, such as Chief Engineer. One key difference between the 0PM 
SECCM Report and the Redstone SECCM Tool is the ability to trace specific 
competencies to grades. This was a challenge to implement into the Redstone SECCM 
Tool and was ultimately dropped in favor of an easier interface. Multiple stakeholders 
noted that it would be beneficial for employees to understand what key competencies are 
missing in order achieve a specific position such as Chief Engineer or Technical Eead. 
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Both of these could be added to the tool with careful changes to the proficiency 
development of the tool. 

9. Recommendation: Printable Report 

Although the tool was considered user-friendly, there was an appetite among the 
stakeholders to output the information into a printable report that could be presented to a 
higher authority. Excel must be formatted to print in a printer-friendly format. No such 
format exists within the tool. 

10. Recommendation: Employee Historical Progression 

In order to track the career progression of an employee over a long period of time, 
there must be a built-in capability to view past competency reports and see the 
progression over time of each competency. This would require a form of record keeping. 
A database system would most likely be the easiest way to implement this capability, 
with the tool sending reports into the database annually. 

11. Recommendation: Additional Graphing Capability 

The users were satisfied with the upgraded graphics capability, but there were 
other desired graphing options still missing. Currently, one may not manipulate the only 
type of graphics available. Each user may have a different preference on how the 
graphics are displayed or filtered, and the ability to modify the graphics would aid the 
user in the creating a view that works best for the situation. One potential use of this is to 
show all employees along with associated competency models, while hiding personal 
information such as names. 
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APPENDIX: NEEDS ANALYSIS RESULTS 


NEEDS ANALYSIS RESULTS TABLES 


Table 15. Primary Stakeholder Quotes 


Stakeholder 

Quotes 

AMRDEC ED: 

“Competencies are not binary,” “Competencies should be based 
on experience,” “9 competencies work for us,” “Competencies 
should be locally managed.” 

PEG M&S 1: 

“Experience gained through multiple job rotations is the most 
important” 

PEG M&S 2: 

“Competency model could be useful however I would invert the 
rows and columns and locally define the needed competencies.” 

PEG M&S 3: 

“Gn the job experience trumps competency models with breadth 
and depth of knowledge important.” 

AMRDEC 

SED: 

“A real competency model must be like any model: It must be 
prescriptive, must meet the needs of the organization, must yield 
quantitative results, must be verifiable/quantifiable, and subject 
to continues improvement.” 

AMRDEC 

WDI: 

“Experience depth first, before you get breadth,” “Understand 
the entire life cycle,” “define level of experience by grade,” 
“locally managed competency models” 

PEG M&S 4: 

“Up front is most important,” “Need an SE degree,” 

“Institutional training,” “All program offices act differently” 

CP-16: 

“Competency model is a good idea but implementation will be 
problematic,” “Existing systems engineers may not meet the new 
requirements.” 

MDA: 

“44 competencies are too many,” “Systems engineers are not 
well defined, should focus on requirements” 
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Table 16. Stakeholder Empathy maps 


Stakeholder 

Feeling Toward 
Competency Model (higher 
is more positive) 

Think and Feel 

AMRDEC ED: 

5 

Multi-level competencies, 
experience based competencies, 
locally managed competencies, 
small number of competencies 

PEG M&S 1: 

4 

experience based competencies 

PEG M&S 2: 

2 

locally managed competencies 

PEG M&S 3: 

5 

experience based competencies 

AMRDEC SED: 

5 

locally managed competencies, 
experience based competencies 

AMRDEC WDI: 

4 

locally managed competencies, 
experience based competencies 

PEG M&S 4: 

3 

locally managed competencies 

CP-16: 

3 

N/A 

MDA: 

3 

small number of competencies 


Table 17. Stakeholder Personas 


Persona 

Stakeholder 

SE is any engineer that works on a system 

AMRDEC WDI 

PEG M&S 1 

PEG M&S 2 

AMRDEC SED 

SE does work as defined in INCGSE HB 

AMRDEC ED 

PEG M&S 3 

PEG M&S 4 

SE is requirements focused 

MDA 

No opinion/impartial 

CP-16 
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CANDIDATE QUESTIONS FOR SECCM CAPSTONE PROJECT 


Specific for Capstone Stakeholders: 

At the Organization Level 

1. Describe the Systems Engineering career field as seen by your organization. What 
are the roles of the Systems Engineers within your organization? 

2. What guidelines or criteria are used within your organization for hiring Systems 
Engineers (entry, mid-level, senior, and lead positions)? What guidelines or 
criteria are used for creating job descriptions for hiring Systems Engineers? 

3. In hiring a systems engineer, is depth of knowledge in a particular competency or 
breadth of understanding across all competencies prioritized? If depth of 
knowledge is prioritized, which competencies does this apply for your 
organization? 

4. Is a Systems Engineering competency model currently applied in your 
organization, and if so, how is it used? What model is used? 

5. How would having a DOD promulgated Systems Engineering Competency Model 
help the manager/hiring manager? What gaps still exist? What changes would 
help your organization? 

6. Would having a defined occupational series for Systems Engineering benefit the 
DOD as a whole, and if so, please explain how? Erom your organizational 
perspective, what are the potential downsides of a DOD Systems Engineering 
occupational series if implemented? 

7. Out of the 44 Critical Competencies, are there any that are unique to Systems 
Engineering only for the Army? 

8. Which of the 44 Critical Competencies stand out as most important for a Systems 
Engineer and why? Are there any natural groupings within those competencies? 

9. Is the Army Systems Engineering Competency Model missing any critical 
competencies? If so, what are they? 

10. Do Systems Engineers across the Army lack any competencies necessary to 
perform the role of systems engineering? If so, what are they? 

11. What percentage of work time does the average Systems Engineer spend doing 
technical work rather than programmatic or managerial work, at the Entry level 
(GS-7 through GS-11), Journeyman level (GS-12, GS-13), GS-14, and GS-15 
levels? 
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12. Imagine for a minute that the SE Competencies were organized into a pyramid 
with “Required at Entry” competencies on the bottom and the highest-level 
competencies on top. What different levels would be in this pyramid? Would this 
product help in defining the position description at multiple grade levels [Entry 
level (GS-7 through GS-11), Journeyman level (GS-12, GS-13), GS-14, and GS- 
15 levels] within your organization? 

13. Suppose a team of 4 Systems Engineers is proposed to work cooperatively on a 
project. Describe how their abilities, skillsets, or competencies would preferably 
differ and align. Now given the competencies in Table 21 of the Army 
Occupational Survey, group the competencies that best represent the differing 
skillsets of the 4 Systems Engineers as desired. 
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SUPPLEMENT ALS 


The following products are packaged with this report or may be obtained by 
contacting the Dudley Knox Library at the Naval Postgraduate School. 

A. REDSTONE SYSTEM ENGINEERING CAREER COMPETENCY 
MODEL TOOL 

A Microsoft Excel-based systems engineer career competency model tool, named 
Redstone SECCM Tool, assists and promotes the creation of, and tracking of, an 
individual or organizational competency model. The tool provides a wizard-based, user- 
friendly interface for creating competency models that are intended to be traceable back 
to the 0PM SECCM competencies. The tool provides flexibility by allowing the user to 
either generate organizational models for dissemination or develop employee competency 
models. The employee models can be used within the local organizations for developing 
training plans 

B. REDSTONE SYSTEM ENGINEERING CAREER COMPETENCY 
MODEL TOOL USER GUIDE 

This user guide is an instruction for using the Redstone Systems Engineering 
Career Competency Model Tool (Redstone SECCM Tool) developed and managed by 
systems engineering students at the Naval Postgraduate School. Although this guide is 
intended to guide the development of a competency model within a specific tool, many of 
the steps and procedures can be applied to any general competency model development 
effort, regardless of tool used. 
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